Penalty Terms for Fault Tolerance

This paper presents penalty terms for fault tolerance enhancement. We argue that the use of conventional learning algorithms leads to networks that have solutions that are non-optimally distributed and hence susceptible to faults. In addition we assert that fault tolerance will become an increasingly important factor in practical applications of neural networks. To address these issues we present a realistic model of hardware error and go on to propose a new method for optimizing fault tolerance via penalty terms. The penalty terms are incorporated into the learning to optimize the network for the smoothness of the solution locus. Such smoothness can be thought of as low average weight saliency and optimally distributed computation. We compare two roughness penalty terms with our previous work with weight-noise. Results from MLPs trained on two problems, one artificial and the other a real world task, show that fault tolerance can be achieved for a realistic fault model via the use of penalty terms.